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Monday,  August  12 

12.30- 13.45  Lunch  in  salons  Artnberg,  Arenberg  castle  Page 

13.45-14.00  Opening  of  the  Workshop 

\ 

Session  1  :  Basic  concepts  of  the  TLS  problem  j  j 

14.00-14.30  Gene  H.  Golub  26 

Basic  principles  of  the  TLS  problem. - 

14.30- 15.30  Sabine  Van  Huffel  42 

The  TLS  problem:  current  state  of  research  and  applications 

^  15.30-16.00  Break 

Session  2  :  Structured  and  constrained  TLS  problems  , 

16.00-16.30  James  Demmel  20 

TLS  with  structured  perturbations 

16.30- 17.00  Bart  De  Moor  21 

Structured  TLS  problems 

17.00-17.30  Theagenis  J.  Abaizoglou  11 

Constrained  TLS  and  superresolution  signal  processing 

17.30- 18.00  James  A.  Cadzow  17 

Constrained  TLS  problem 

19.00  Dinner  in  restaurant  “De  Oude  Kantien” 


r 


L 
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Tuesday,  August  13 

^ession  3  :  Some  special  TLS  problems  ^ 

09.00-09.30  Musheng  Wei 

The  TLS  problem  with  more  than  one  solution:  solutions, 
perturbation  theory  and  relations  with  the  LS  problem 

y 

09.30- 10. GO  James  R.  Bunch  and  Ricardfi^iD.  Fierro 

Collinearity  and  the  cyie^mensional  TLS  problem 

10.00-10.30  Trevori-Jici^e,  Tom  Duchamp  and  Werner  Siuetzle 
'“Principal  curves 


10.30-11.00  Break 


Session  4  :  TLS  algorithms 

11.00-11.30  Gilbert  W.  Stewart 

Updating  rank  revealing  URV  decompositions 

11.30-12.00  Sabine  Van  Huffel  and  Hongyuan  Zha 

.4n  efficient  TLS  algorithm  based  on  a  rank- revealing 
two-sided  orthogonal  decomposition 

12.00-12.30  ^-arc  Moonen 

Algorithms  and  architectures  for  recursive  total  least 
squares  estimation 


Page 

46 


29 


38 

43 


33 


12.30-14.00  Lunch  in  salons  Arenberg,  Arenberg  castle 

't  ‘t  .  .  .  •  ' 

Session  5  ;  Errorstimvariables  and  statistical  properties,  part  1  , 

j 

14.00-14.30  Wayne  A.  Puller 

Properties  of  estimators  for  the  errors-in-variables  model 
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25 


14.30-15.00  Hugo  Van  hamme  and  Rik  Pintelon  41 

Asymptotic  properties  of  a  class  of  regression-type  estimators 

15.00-15.30  John  Van  Ness  44 

Robust  estimation  in  measurement  error  models 
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15.30- 16.00  Break 

Session  6  :  Errors-in*varisibles  and  statistical  properties,  part  2  ^ 

16.00-16.30  Chi-Lun  Cheng  18 

TLS  and  errors-in- variables  regression 

16.30- 17.00  Larry  P.  Ammann  13 

Aignrithms  for  robust  M-estimation  of  cov'ariance  eigen-structure 

17.30- 19.30  Guided  visit  through  Leuven:  beguinage,  city  hall,  university 

20.00  Dinner  in  restaurant  “De  Oude  Kantien” 
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Wednesday,  August  14 

Session  7  :  Errors/in-variables  and  system  identification 

09.00-09.30  Torsten  Soderstrorn 

Some  aspects  on  treating  measurement  errors  in 
system  identification 


09.30-10.00  Roberto  P.  Guidorzi 

Errors-in-variables  identification  and  model  uniqueness 

10.00-10.30  Adina  Stoian 

Comparativft.4>aFf'orTfiance  study  of  least  squares  and  TLS 
Yufe-Wriiker  estimates  of  autoregressive  parameters 

10.30-11.00  Break 


Session  8  :  TLS  applications  in  signal  processing 


J 


11.00-11.30  Donald  W.  Tufts  and  Abhijit  A.  Shah 

Signal  enhancement  motivated  by  TLS  and  Jihear  prediction 


11.30-12.00  Richard  Roy 

Reed  world  applications  of  TLS:  Direction-finding  and  system 
identification 


12.00-12.30  ^Michael  P.  Clark  and  Louis  L.  Scharf 

Maximum  likelihood  parameter  estimation  for  array  processing 

12.30- 14.00 
Session  9  ; 

14.00-14..30 

14.30- 1.5.00  Paul  T.  Boggs 

Orthogonal  distance  regression 

15.00-15.30  Janet  E.  Rogers 

ODRPACK:  software  for  orthogonal  distance  regression 


Lunch  in  salons  Arenberg,  Arenberg  castle  y  .1 

Nonlinear  TLS  problems  ^  - - 

Yasuo  Amemiya 

Parameter  estimation  in  nonlinear  errors-in-variables  problems 


lo. 30-16. 00  Break 


:  Other  TLS  applications  . 

J  ' 

George  W’.  Fisher  23 

TLS  methods  ot  analyzing  mineral  assemblages  and  reactions 
in  metamorphic  rocks 


16.30-17.00 

Jae  H.  Lee 

The  use  of  TLS  in  an  oceanographic  problem 

30 

17.30-17.30 

Mary  B.  Seasholtz  and  Bruce  R.  Kowalski 

Tensorial  calibration  in  chemistry 

36 

17.30-18.00 

Trevor  Hastie,  Eyal  Kishon,  Malcolm  Clark  and  Jason  Fan 

A  model  for  signature  verification 

28 

18.00-19.00 

Demonstrations 

19.30 

Banquet 

Session  10 

16.00-16.30 
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Thursday,  August  15 


Session  11  : 


Extensions  of  TLS  to  infinite  dimensional  spaces 
and  other  norms  , 


09.00-09.30  G.  Alis'air  Watson 

Total  C'hebysh'-v  app.  ,  ^ination 


09.30-10.00  A.  5.  Arun 

Infinite  dimensional  TLS  problems 


10.00-10.30  Jfan-Fran(;ois  Maitre 

TLS  and  acceptable  generalized  solution  to  a  problem. 
Theoretical  results  for  general  norms 


10.30-11.00  Bnak- 


11.00-12.1.5 


12.15-12.30 


12.30-14.00 

14.00 


Open  problems  and  discussion 

Closing  of  the  Workshop 

Lunch  in  salons  Arenberg,  Arenberg  castle 

End. 


f^age 
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31 
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Constrained  Total  Least  Squares  Problem 
and  Superresolution  Signal  Processing. 


Theagenis  J.  Abatzoglou 
Lockheed  Missiles  &  Space,  Inc., 

Research  &  Development  0/91-50,  B251, 

3251  Hanover  Street,  Palo  Alto,  CA  94304-1191,  U.S.A. 


Constrained  Total  Least  Squares  (CTLS)  is  a  generalized  version  of  Total  Least  Squarer 
wiiere  t  :.?  noise  perturbations  to  the  coefficients  of  a  linear  system  of  equations  are  allowed 
to  satisfy  linear  relations.  Theoretical  properties  of  the  CTLS  solution  are  presented  which 
i..clude  its  equivalence  to  a  Maximum  Likelihood  estimator,  reduction  of  the  solution  to 
an  unconstrained  Iciver  dimensionality  problem,  perturbation  analysis  and  numerical  algo¬ 
rithms. 

.Applications  of  CTLS  to  signal  processing  are  reabzed  by  imposing  a  linear  model  to 
data  sa,mples.  Linear  prediction  is  a  good  model  ^or  harmonic  signals;  applications  include: 
frequency  estimation  (cne-  or  two-dimensional),  direction  finding,  extraction  of  resonances 
of  objects,  etc. 
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Parameter  Estimation  in  Nonlinear 
Errors-in- Variables  Problems. 

Yasuo  Amemiya 

Department  of  Statistics,  Iowa  State  University, 
102G  Snedecor  Hall,  Ames,  Iowa  50010,  U.S.A. 


The  functioned  errors-in-variables  model  assumes  that  observations  are  the  sums  of  un¬ 
derlying  true  \'alues  and  measurement  errors,  and  that  the  true  values  are  fixed  constants 
satisfying  some  relationship.  The  case  where  the  relationship  is  specified  by  a  nonlinear 
parametric  function  is  considered.  The  maximum  likelihood  estimator  of  the  parameter 
under  the  assumption  of  normal  measurement  errors  is  a  nonlinear  total  least  squares  esti¬ 
mator.  It  is  shown  that  such  an  estimator  has  a  nonnegligible  bias  due  to  the  nonlinearity 
and  measurement  errors.  .An  estimator  that  does  not  have  such  a  bias  is  suggested.  The¬ 
oretical  and  numerical  results  supporting  the  superiority  of  the  bias-adjusted  estimator 
relative  to  the  ma.ximum  Likelihood  estimator  are  presented. 
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Algorithms  for  Robust  M-Estimation  of 
Covariance  Eigen-structure. 

L  arry  P.  Ammann 
Programs  in  Mathematical  Sciences, 

University  of  Texas  at  Dallas, 

Richardson,  TX  75080,  U.S.A. 

Robust  location  and  covariance  estimators  are  developed  via  general  M-estimation  for 
covariance  matrix  eigenvectors  and  eigenvaJues,  which  results  in  a  generalization  of  pro¬ 
jection  pursuit.  The  solution  to  this  M-estimation  problem  is  obtained  by  transforming  it 
into  a  series  of  robust  regression  problems-  The  motivation  for  this  transformation  is  the 
interpretation  of  the  singular  value  decomposition  as  an  iteration  of  two  steps  —  a  least 
squares  regression  fit  of  the  data  matrix  followed  by  a  rotation  to  the  regression  hyper¬ 
planes.  An  algorithm  to  obtciin  this  solution  is  presented,  cdong  with  results  of  a  Monte 
Carlo  study  and  examples  of  its  application. 

This  algorithm  is  derived  by  efficiently  replacing  least  squares  regression  with  robust  re¬ 
gression,  which  then  produces  a  broad  class  of  new  robust  methods  for  multivariate  data. 
Some  results  in  numerical  linear  algebra  are  presented  which  describe  the  relationships 
among  least  squares  regression,  the  QR  decomposition,  and  the  singular  value  decompo¬ 
sition.  These  results  are  utilized  in  the  algorithm  to  solve  the  M-estimation  problem  so 
that  it  can  be  built  upon  standard,  commonly  available,  numerical  linear  algebra  routines. 
In  addition,  it  is  shown  how  the  output  of  this  algorithm  can  be  used  to  search  numeri¬ 
cally  for  multivariate  outliers.  This  i"  especiedly  useful  in  exploratory  data  analysis  with 
high-dimensioned  data  where  graphical  techniques  are  difficult  to  implement.  Since  the 
algorithm  computes  robust  estimates  of  the  eigenvectors  and  eigenvalues  of  the  covariance 
matrix,  it  can  be  used  as  a  basis  for  other  multivariate  methods  such  as  signal  subspace 
ealimation  and  direction  of  arrival  problems,  errors-in-variables  regression,  discriirunant 
analysis,  and  principal  components.  The  examples  of  its  application  given  here  include 
some  data  sets  which  have  been  examined  by  other  methods  for  comparison  purposes,  and 
an  application  to  a  large  data  set  taken  from  the  Landsat  satellite. 


Infinite  Dimensional  Total  Least  Squares 

Problems. 


K.  S.  Arun 

Coordinated  Science  Laboratory, 
University  of  Illinois, 

llOl  West  Springfield  Avenue, 
Urbana- Champaign,  IL  61801,  U.S.A. 


In  this  lecture,  we  will  formulate  and  solve  infinite  dinieiisional  versions  of  the  generic 
and  non-generic  total  least  squares  problems,  and  apply  it  to  problems  such  as  decon¬ 
volution,  linear-min-variance  estimation,  system  identification,  and  Kalman  and  Wiener 
filtering. 
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Orthogonal  Distance  Regression 


Paul  T.  Boggs 

National  Institute  of  Standards  and  Technology  (NIST), 

Applied  and  Computational  Mathematics  Division, 
Gaithersburg,  MD  .20899,  U.S.A. 


We  describe  the  Orthogonal  Distance  Regression  problem  (ODR),  i.e.,  the  problem  of 
fitting  a  function  to  a  given  set  of  data,  {x{i),y(i)),  i  =  1, . . .  ,7i,  where  errors  are  assumed 
to  exist  in  both  the  x-values  as  well  as  in  the  j/-vaJues.  This  is  in  contrast  to  ordinary  least 
squares  (OLS)  where  all  of  the  errors  are  assumed  to  exist  in  the  y- values.  We  first  precisely 
define  the  (ODR)  problem,  and  then  briefly  describe  a  stable  and  efficient  algorithm  for  its 
solution;  the  algorithm  and  its  implementation  in  the  software  library  ODRPACK,  as  well 
as  examples  that  have  arisen  at  NIST,  will  be  discussed  in  detail  by  J.  Rogers.  Here,  we 
will  concentrate  on  simulation  studies  that  compare  ODR  to  OLS  in  a  controlled  setting. 
In  particular,  we  compare  ODR  with  OLS  under  the  assumption  that  the  ratio  of  the  errors 
in  y  to  the  errors  in  x  is  known.  Several  measures  of  performance  are  considered.  We  then 
relax  this  assumption  and  compare  the  performance  of  ODR  when  the  ratio  is  not  known 
precisely.  Finally,  we  discuss  the  computation  and  use  of  the  asymptotic  covariance  matrix. 
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Collinearity  and  the  One-Dimensional  Total 
Least  Squares  Problem. 

James  R.  Bunch  and  Ricardo  D.  Fierro 
Department  of  Mathematics, 

University  of  California  a'  San  Diego, 

La  Jolla,  CA  92093-01  Iz,  U.S.A. 


The  numerical  effects  of  collinearity  in  the  columns  of  A  on  the  TLS  estimator  will  be 
considered.  The  TLS  problem  can  become  ill-conditioned  and  a  regularization  technique 
is  employed  to  stabilize  the  solution  by  increasing  the  dimension  of  the  solution  space.  A 
condition  estimator  for  the  new,  but  related,  problem  is  proposed;  it  coincides  with  the 
condition  number  in  the  generic  case.  An  approximation  theorem  will  be  presented  which 
more  formally  specifies  the  near-intimate  relationship  between  SVD  components  of  A  and 
[A,  6]  when  the  last  component  of  the  jth  right  singular  vector  of  [A,  6],  is  “small”. 

For  example,  this  arises  when  the  jth  singular  value  of  (A,  6]  is  nonpredictive. 

The  smallest  singular  values  of  [A,  6]  inherited  from  A  may  identify  collinearities  that 
are  all  predictive,  or  all  nonpredictive,  or  a  combination  of  both.  We  will  present  a  theorem 
which  implies  that  one  should  deduce  a  minimum  norm  solution  in  a  space  that  includes 
the  singular  vectors  associated  with  the  collinearities  inherited  from  A,  regardless  of  their 
predictive  value.  Intuitively,  when  the  clustered  singular  values  are  included,  the  solution 
space  is  more  “stable”. 

We  will  compare  the  TLS  solution  to  the  Truncated  LS  solution  of  Bx  =  6,  where  B  is 
the  nearest  rank  r  approximation  to  A.  We  will  discuss  how  the  regularized  TLS  solution 
is  also  the  unique  minimum  norm  solution  to  a  particular  LS  problem  that  has  a  higher 
condition  number  than  the  Truncated  LS  problem  of  the  same  dimension.  Although  the 
matrix  can  differ  in  its  singular  values  and  singular  vectors  from  those  of  B,  the  singular 
triplets  associated  with  the  largest  singular  values  are  usually  close  to  the  corresponding 
ones  of  B. 
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Constrained  Total  Least  Squares  Problem 


James  A.  Cadzow 

Department  of  Electrical  Engineering, 

Vanderbilt  University, 

P.O.Box  6080,  STA.  B,  Nashville,  Tennessee  31235,  U.S.A. 


In  the  standard  total  least  squares  problem,  one  is  given  a  system  of  inconsistent  linear 
equations 

e  =  Ax  —  b 

in  which  the  matrix  A  and  vector  6  are  subject  to  error.  It  is  now  desired  to  perturb  the 
system  matrix  A  to  A  E  and  the  vector  6  to  6  +  r  so  that 

1.  [A  +  E]x  =  6  +  r  is  consistent 

2.  of  all  such  modifications,  the  perturbations  used  have  minimum  Euclidean  norm. 


In  many  applications,  the  underlying  noise  free  matrix  [A;  b]  will  possess  theoretical  proper¬ 
ties  such  as  having  a  specific  structure  (e.g.,  Toeplitz)  and  a  given  rank.  In  order  to  retain 
these  properties,  the  perturbations  E  and  r  need  be  appropriately  constradned.  In  this 
presentation,  an  algorithm  is  presented  for  achieving  the  minimum  Euclidean  perturbation 
so  that  the  perturbed  system  of  equations  possesses  the  required  properties. 
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Total  Least  Squares  and  Errors-in- Variables 

Regression. 


Chi-Lun  Cheng 

Institute  of  Statistical  Science, 
Academia  Sinica,  Taipei,  Taiwan,  R.O.C. 


This  talk  surveys  the  problem  of  total  least  squares  from  a  theoretically  statistical 
viewpoint.  It  includes  the  errors-in- variables  model  and  principal  component  analysis 
which  are  numerically  equivalent  to  total  least  squares  under  certain  assumptions.  We  edso 
discuss  the  robust  version  of  the  statistical  models  corresponding  to  total  lea^t  squares. 


Maximum  Likelihood  Parameter  Estimation 
for  Array  Processing. 

Michael  P.  Clark  and  Louis  L.  Scharf 
Department  of  Electrical  and  Computer  Engineering^ 

University  of  Colorado, 

Campus  Box  425,  Boulder,  CO  80309-0425,  U.S.A. 


We  use  the  data  collected  by  a  multisensor  array  to  identify  the  Directions  Of  Arrival 
(DOA),  frequency  and  amplitude  of  individual  point  sources  in  the  far-held.  Many  methods 
for  solving  this  problem  have  been  developed.  However,  most  of  these  methods  work  only 
for  sources  with  the  same  center  frequency.  Those  which  do  allow  arbitrary  frequencies 
can  predict  both  the  frequencies  and  the  DOAs.  However,  these  methods  have  difficulty 
associating  (pairing)  the  frequencies  and  DOAs.  We  propose  an  edgorithm,  based  on  the 
principle  of  maximum  likelihood,  which  solves  the  “pairing”  problem  for  arrays  with  linear 
geometries.  The  method  works  by  identifying  a  matrix  which  characterizes  the  orthogonal 
subspace.  This  matrix  has  two  parts.  The  first  of  these  is  based  on  a  temporal,  whitening 
polynomial.  The  second  uses  an  interpolating  polynomial  which  predicts  spatial  modes 
from  temporal  modes.  The  method  works  by  solving  for  the  parameters  which  chararc- 
terize  the  orthogonal  subspace,  making  it  a  subspace  method.  The  orthogonal  subspace 
is  identified  by  any  of  several  methods,  including  least  squares  and  total  least  squares. 
Once  the  parameters  of  the  orthogonal  subspace  have  been  estimated,  the  temporal  poly¬ 
nomial  is  rooted  to  obtain  the  temporeil  modes.  These  modes  are  then  plugged  into  an 
interpolating  polynomial  to  generate  the  corresponding  spatial  modes.  Simple  geometrical 
considerations  then  yield  the  DOA  estimates. 
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Total  Least  Squares  with  Structured 
Perturbations. 


James  Demmel 

University  of  California  at  Berkeley, 

Computer  Science  Division, 

Evans  Hall  513,  Berkeley,  CA  94720,  U.S.A. 


The  classical  TLS  solution  of  AX  =  B  involves  using  the  SVD  to  find  a  nearest  rank 
deficient  matrix  to  [A,B\.  This  treats  errors  in  all  entries  of  A  and  B  equally.  One  often 
wants  to  impose  more  structure  on  perturbations  in  A  and  B.  Previous  work  has  imposed 
row  and  column  sccdings  or  block  structures  on  the  perturbations.  We  discuss  imposing  an 
independent  error  scaling  on  each  entry  of  A  and  5;  i.e.  the  error  norm  is  not  given  by  a 
ball  but  by  an  arbitrary  rectangular  parallelepiped.  We  discuss  recent  results  of  Rohn  and 
Poljak  who  give  a  solution  for  a  certain  special  case,  and  show  that  even  in  this  special  case 
the  problem  is  NP-complete.  Recent  work  on  approximating  this  NP-complcte  solution  as 
well  as  open  problems  are  discussed. 
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structured  Total  Least  Squares  Problems 


Bart  De  Moor^ 

Katholieke  Universiteit  Leuven, 

ESAT  Laboratory,  Department  of  Electrical  Engineering, 
Kardinaal  Mercierlaan  94,  B-3001  Heverlee,  Belgium 


Given  a  real  matrix  A  t  with  p  >  q  and  rank(A)  =  q.  Let  A  be  decomposed  as 
A  =  F  +  M  where  F  contains  the  elements  that  cannot  be  modified  and  M  contains  the 
elements  to  be  modified.  The  zero  patterns  in  F  and  A/  are  complementary:  fjmij  =  0. 
The  structured  total  least  squares  problem  is  the  problem  of  finding  a  rank  deficient  matrix 
B  —  F  +  N  so  that  ||A/  —  .(V||^  is  minimized  and  the  zero  pattern  of  N  is  complementary 
to  that  of  F:  nijfij  =  0. 


When  F  =  0,  the  problem  is  known  as  the  total  least  squares  problem  [4]  and  is  solved 
using  the  singular  value  decomposition  [3].  Solutions  for  some  specially  structured  matrices 
A  have  been  obtained,  such  as  a  matrix  A  with  ’exact’  columns  (i.e.  where  F  is  of  the  form 
F  =  [  Fi  0  ] )  [5]  or  when  only  the  lower  right  hand  corner  block  of  a  four  block  partitioned 
matrix  A  can  be  modified  [1]  [2].  It  remains  unsolved  however  in  full  generality. 

We  provide  some  more  insight  by  showing  that  the  solution  is  equivalent  with  solving  a 
special  type  of  SVD  completion  problem. 

We  also  derive  a  remarkable  correspondence  between  the  structured  TLS  problem  where 
.4  is  four  block  partitioned  and  three  out  of  four  blocks  can  be  modified,  and  the  solution  to 
the  following  total  least  squares  problem  with  three  matrices:  Given  A  G  B  6 

and  C  6  3?**".  Find  X  €  so  that  the  object  function  ||i4  — F!|^  +  ||F  — Q||f +||C  — Fljf- 
is  minimized  subject  to  PXQ  =  R.  A  solution  for  the  scalar  case  where  A  is  a  column 
vector  and  B  a  row  vector  has  been  obtained  recently  and  will  probably  be  generalized  by 
the  time  of  the  workshop. 
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Total  Least  Squares  Methods  of  Analyzing 
Mineral  Assemblages  and  Reactions  in 
Metamorphic  Rocks. 

George  W.  Fisher 

Department  of  Earth  and  Planetary  Sciences, 

The  Johns  Hopkins  University, 

Baltimore,  Maryland  21218,  U.S.A. 


Interpretation  of  minercd  assemblages  in  metamorphic  rocks  depends  upon  determining 

1.  whether  or  not  individual  assemblages  could  represent  equilibrium  states; 

2.  whether  or  not  pairs  of  assemblages  could  have  equilibrated  under  the  same  pressure, 
temperature  conditions;  and 

3.  the  nature  of  the  reactions  relating  assemblages  equilibrated  under  different  condi¬ 
tions. 

Traditional  methods  of  analysis  rely  on  graphical  projection  methods  which  are  simple  to 
use,  but  can  not  deal  effectively  with  assemblages  involving  variation  in  more  than  three 
components.  In  multicomponent  assemblages,  these  questions  are  best  approached  by  in¬ 
vestigating  the  rank,  composition  space  (range)  and  reaction  space  (null  space)  of  matrices 
in  which  the  phases  of  an  assemblage  are  represented  by  columns  and  the  components  by 
rows. 

Total  Least  Squares  (TLS)  methods  based  on  the  singular  value  decomposition  provide 
a  simple  and  elegant  way  of  finding  a  model  matrix  of  specified  rank  which  approximates 
the  observed  matrix  within  analytical  uncertainty.  Examination  of  the  elements  of  the 
null  space  (or  reaction  space)  associated  with  that  matrix  provide  unambiguous  criteria 
for  identifying  possible  equilibrium  assemblages  and  for  determining  whether  or  not  two 
assemblages  can  have  equilibrated  under  the  same  external  conditions:  an  empty  null 
space  reflects  a  possible  equilibrium  assemblage;  a  null  space  containing  reactions  in  which 
the  minereds  of  two  assemblages  have  opposite  signs  indicates  that  the  two  assemblages 
intersect  in  composition  space  and  therefore  can  not  have  equilibrated  under  the  same 
conditions;  a  null  space  lacking  any  such  reactions  indicates  that  the  two  assemblages  do 
not  intersect,  and  so  may  have  equilibrated  under  the  same  conditions. 

I  have  developed  a  set  of  easy-to-use  computer  programs  which  perform  these  opera¬ 
tions.  Trial  runs  using  analyzed  mineral  compositions  from  three  outcrops  spanning  the 
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lower  sillimanite  zone  of  northwestern  Maine,  described  by  C.T.  Foster  (1977,  Am.  Min¬ 
eralogist)  illustrate  the  power  of  the  method.  All  assemblages  contain  the  seven  phases 
sillimanite.  staurolite,  biotite,  garnet,  muscovite,  quartz,  plagioclase  and  ilmenite.  .411 
assemblages  e.xamined  have  a  rank  of  seven,  requiring  seven  components,  and  probably 
represent  divariant  equilibrium.  Assemblages  from  individual  outcrops  show  no  significant 
intersections,  while  those  from  outcrops  differing  in  temperature  by  as  little  as  6’  C  show 
systematic  intersections  reflecting  divariant  reactions. 
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Properties  of  Estimators  for  the 
Errors-in-variables  Model. 


Wayne  /i.  Puller 

Department  of  Statistics,  Iowa  State  University, 
Snedecor  Hall,  Ames,  Iowa  50010,  U.S.A. 


The  method  of  total  least  squares  can  be  used  to  produce  estimates  of  the  oarameters 
of  a  statistical  model.  For  the  linear  model,  the  properties  of  the  estimators  can  be  derived 
under  relatively  weak  assumptions.  We  give  conditions  sufficient  for  the  estimators  to 
be  statistically  consistent.  Also,  an  estimator  of  the  covariance  matrix  of  the  limiting 
distribution  of  the  estimated  parameter  vector  is  presented.  The  estimation  procedures 
have  been  implemented  in  a  personal  computer  program. 
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Basic  Principles  of  the  Total  Least  Squares 

Problem. 

Gene  H.  Golub 

Department  of  Computer  Sciences, 

Stanford  University, 

Stanford,  CA  94305,  US. A. 


We  review  some  of  the  basic  ideas  and  principles  involved  in  the  development  of  Total 
Least  Squares. 
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Errors-in- variables  Identification  and  Model 

Uniqueness. 

Roberto  P.  Guidorzi 

Diparttimento  di  Elettronica,  Informatica  e  Sistemistica, 
Universitd  di  Bologna, 

Viale  del  Risorgimento  2,  40136  Bologna,  Italy 


The  problem  of  deriving  possible  linear  relations  from  data  affected  by  additive  noise 
hcis  received  remarkable  attention  in  recent  years  particularly  regarding  the  assumptions 
(“prejudices”)  behind  the  procedures  leading  to  unique  models.  Unlike  the  many  ap¬ 
proaches  leading  to  unique  solutions  (Least  Squares,  Maximum  Likelihood,  etc.)  the  Frisch 
scheme,  belonging  to  the  family  of  Errors-in- Variables  (EV)  schemes,  leads  to  a  whole  fam¬ 
ily  of  models  compatible  with  a  set  of  noisy  data  and  is  considered  as  only  mildly  affected 
by  prejudices. 

This  talk  discusses  the  loci  of  all  possible  solutions,  in  the  noise  space,  under  the  as¬ 
sumptions  of  the  Frisch  scheme  and  shows  how  it  can  be  possible,  when  different  sets  of 
noisy  data  are  available,  to  obtain  the  unique  model  behind  the  data  and  also  to  derive  the 
actual  amount  of  noise  on  the  data  when  the  noiseless  data  are  linked  by  a  single  linear 
relation.  The  more  general  EV  case  of  non-independent  additive  noises  is  then  consid¬ 
ered  and  it  is  shown  how,  under  the  same  assumptions,  it  is  still  possible  to  obtain  the 
unique  set  of  parameters  linking  the  noiseless  data  and  the  whole  family  of  compatible  noise 
covariance  matrices  which  is  defined,  in  this  case,  by  the  infinite  elements  of  a  linear  variety. 

Finally,  the  previous  considerations,  which  regard  algebraic  systems,  are  extended  to 
dynamic  systems;  in  this  case,  the  dimensions  of  the  noise  and  parameter  spaces  are  no 
longer  equal  and  also  the  order  of  the  model  must  be  estimated.  Also  in  this  case,  proce¬ 
dures  can  be  developed  for  a  consistent  estimation  of  the  model  order  and  the  parameters 
and  for  the  estimation  of  the  noise  covariance  matrix. 
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A  Model  for  Signature  Verification 


Trevor  Hastie,  Eyal  Kishon,  Malcolm  Clark  and  Jeison  Fan 
AT&T  Bell  Laboratories, 

600  Mountain  Avenue,  Murray  Hill,  NJ  019H,  U.S.A. 


We  propose  a  statisticed  model  for  dynamic  signature  verification  by  computer.  Our 
model  recognizes  that  repeated  signatures  by  the  “owner”  are  similar  but  not  identical. 
Our  model  consists  of  a  template  signature  for  each  individual,  and  several  factors  which 
allow  for  variations  in  each  rendition  of  this  template.  These  variations  include  the  speed 
of  writing,  as  well  as  slowly  varying  affine  transformations  such  as  size,  rotation  and  shear. 
The  estimated  template  represents  the  “mean”  of  a  sample  of  signatures  from  an  individual, 
and  the  variations  in  the  factors  can  be  used  to  establish  several  measures  of  “variance”. 
These  quantitative  measures  are  essential  for  reliable  signature  verification. 

Estimating  the  template  consists  of  two  essential  steps: 

•  Aligning  or  “time-warping”  the  set  of  realized  signatures  for  an  individual,  and  seg¬ 
menting  cdl  the  signatures  into  “letters”, 

•  given  the  alignment  and  segmentation,  estimating  the  average  shape  of  the  signatures 
using  piece-wise  affine-invariant  TLS. 

A  new  signature  is  verified  by  comparing  it  with  the  template  signature.  It  will  be  judged 
fraudulant  if  its  shape  is  wrong,  or  if  the  speed  signal  with  which  it  wets  rendered  does 
not  match  the  model.  In  this  talk  I  describe  the  model  and  its  estimation,  illustrating  the 
procedures  with  several  examples. 
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Principal  Curves 


Trevor  Hastie 
AT&T  Bell  Laboratories, 

600  Mountain  Avenue,  Murray  Hill,  NJ  07974,  U.S.A. 

Tom  Duchamp 
Department  of  Mathematics, 

University  of  Washington, 

Seattle,  WA  98195,  U.S.A. 

Werner  Stuetzle 
Department  of  Statistics, 

University  of  Washington, 

Seattle,  WA  98195,  U.S.A. 


Principal  curves  are  smooth  one-dimensional  curves  that  pass  through  the  middle  of  a 
p-dimensional  dataset,  providing  a  nonlinear  summary  of  the  data.  They  are  nonparamet- 
ric,  and  their  shape  is  suggested  by  the  data.  In  this  talk  I  will  define  principal  curves, 
discuss  algorithms  for  their  construction,  present  theoretical  results,  and  illustrate  their 
application. 
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The  Use  of  Total  Least  Squares  in  an 
Oceanographic  Problem. 


Jae  Hak  Lee 

Department  of  geology  and  geophysics, 

Yale  University, 

P.O.Box  6666,  New  Haven,  Connecticut  06511,  U.S.A. 


An  appealing  property  of  the  total  least  squares  (TLS)  solution  is  its  invariance  to 
some  c&ses  of  transformations  of  unknowns.  That  is,  for  an  overdetermined  set  of  linear 
equations  Ax=b  where  A  is  the  coefficient  matrix,  b  is  the  data  vector  and  x  is  the  vec¬ 
tor  of  unknowns,  the  TLS  solution  is  not  affected  by  the  rotation  of  the  space  of  A  and 
b  or  a  reciprocal  transformation  of  unknowns.  The  latter  case  is  used  to  treat  a  special 
problem  which  has  no  inhomogeneous  term  (i.e.,  b=0)  but  with  a  priori  information.  It 
is  shown  that,  unlike  the  ordinary  least  squares  method,  the  TLS  method  yields  a  unique 
set  of  solutions.  As  an  application  of  this  property  to  real  situations,  the  TLS  method  is 
applied  to  an  oceanographic  inverse  problem  which  is  to  determine  velocities  and  diffusion 
coefficients  from  known  tracer  (temperature  and  salinity  here)  distributions  based  on  the 
steady-state  advective-diffusive  equation  and  mass  continuity.  The  derived  diffusivities 
lead  to  conclusions  that  are  consistent  with  the  known  physics.  This  indicates  that  the 
application  of  TLS  is  quite  successful. 

We  review  the  invariance  of  the  TLS  solution  by  using  a  geometric  interpretation  of 
the  line  fitting  problem  and  report  the  results  of  this  successful  application  of  the  TLS 
method. 
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Total  Least  Squares  and  Acceptable/ 
Generalized  Solution  to  a  Problem. 
Theoretical  Results  for  General  Norms 


Jean- Francois  Maitre 

Ecole  centrale  de  Lyon, 

Departement  de  Mathematiques, 
B.P.  163,  69131  Ecully  cedex,  Prance 


We  consider  a  problem  in  the  form 

X  e  E,  P{x)  =  0,  (1) 

where  P  is  a  mapping  from  a  set  P  to  a  linear  space  F  and  a  normed  linear  subspace  A 
of  F^.  For  each  x  &  E,  we  define 

e(x)  =  inf{||^Pll;^P  6  A,{P  +  6P)(x)  =  0}.  (2) 

We  say  that  x  is  an  £-solution  to  (1)  if  €  «4  exists  with 

iP  +  6P){x)  =  0,  6P<e,  (3) 

which  is  equivaJent  to  e{x)  <  e. 

We  say  that  Iq  is  a  generalized  solution  to  (1)  if 

e(xo)  =  inin{e(x);  I  t  P}.  (4) 

The  quantity  e(i),  given  by(2),  is  equal  to  ||P(x)||j,,  where  the  semi-norm  ||.||»  is  defined 
by: 

||2/iU  =  inf{||M||;Me  AM(x)  =  y}.  (5) 

We  give  conditions  of  existence  and  properties  of  ||.||».  When  E  and  F  are  normed 
linear  spaces,  A  consists  of  linear  mappings  and  ||.|lj.  is  a  norm  for  a;  7^  0,  we  have  the 
equivalence  : 

Vx  €  P  \  {0},y  e  F,  ||y|U  =  l!y||/||x||  ^^xeE,f^  F',  ||/||:  =  l|x||||/i|  (6) 

and  C{E,F)  is  the  greatest  A  for  which  the  relations  in  (6)  are  right. 

In  the  case  of  a  linear  problem 

Ax-b  =  0,  (7) 
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that  is  P(i)  =  Ax  —  b,  and  special  choices  of  the  norm  of  A  satisfying  (6),  we  have 


e(i)  =  life- yli|l/|l(i,  1)11  (8) 

which  corresponds  for  the  euclidean  norm  to  the  Total  least  Squares  problem.  When  A  is 
limited  to  perturbations  (0,fefe)  we  have  e(x)  =  ||fc  —  Azjl  (Least  Squares),  and  when  A  is 
limited  to  perturbations  (feA,0)  we  have  e(x)  —  ||fe  —  Ai||/l|a:|l. 
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Algorithms  and  Architectures  for  Recursive 
Total  Least  Squares  Estimation. 


Marc  Moonen^ 

Katholieke  Universiteit  Leuven, 

ESAT  Laboratory,  Department  of  Electrical  Engineering, 
Kardinaal  Mercierlaan  94,  B-3001  Heverlee,  Belgium 


Total  least  squares  parameter  estimation  is  an  alternative  to  least  squares  estimation, 
2ilthough  much  less  used  for  instance  in  adaptive  filtering.  The  absence  of  efficient  recursive 
algorithms  here  h2is  apparently  been  a  motive  for  using  Recursive  Least  Squares  (RLS) 
even  in  those  applications  where  typically,  one  would  better  use  the  TLS  technique.  Our 
aim  is  therefore  to  derive  a  similar  recursive  total  lecist  squares  (RTLS)  algorithm,  which 
is  also  amenable  to  parallel  implementation.  It  turns  out  that  a  RTLS  algorithm  C2m 
be  constructed,  based  on  inverse  iteration,  which  very  much  resembles  the  original  RLS 
algorithm.  Algorithmic  and  architectural  considerations  for  RLS  then  straightforwardly 
carry  over  to  the  RTLS  ca^e. 


^Senioi  Research  Assistant  of  the  Belgian  N.F.W.O.  (National  Fund  for  Scientific  Research) 
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ODRPACK:  Software  for  Orthogonal 
Distance  Regression. 

Janet  E.  Rogers 

National  Institute  of  Standards  and  Technology  (NIST), 
Applied  and  Comj.  utational  Mathematics  Division, 

325  Broadway,  Boulder,  Colorado  80303'3328,  U.S.A. 


We  will  examine  the  OrthogonaJ  Distance  Regression  (ODR)  technique  for  fitting  data 
to  a  general  nonlinear  model  when  there  are  errors  in  all  of  the  variables  {x{i),y{i)),  i  — 
1, . . .  ,n.  ODR  solves  this  extended  least  squares  problem  by  minimizing  the  sum  of  the 
squared  orthogonal  distances  between  each  data  point  and  the  curve  described  by  the 
model  equation.  The  number  of  unknowns  involved  in  the  ODR  problem  is  the  number  of 
model  parameters  plus  the  number  of  variables  observed  with  measurement  error,  often  a 
very  large  number.  By  exploiting  the  structure  of  the  problem,  however,  we  have  devel¬ 
oped  a  stable  and  efficient  trust  region  Levenberg-Marquardt  algorithm  with  a  work  per 
step  bound  equal  to  that  required  for  the  Levenberg-Marquardt  method  for  ordinary  least 
squares  (OLS),  where  the  number  of  unknowns  is  simply  the  number  of  model  parameters. 

We  will  describe  our  ODR  algorithm,  and  also  the  use  and  features  of  the  software  li¬ 
brary  ODRPACK  that  is  an  implementation  of  it.  ODRPACK  cein  be  used  to  solve  explicit, 
implicit  and  multiresponse  ODR  problems.  Our  discussion  will  include  several  examples 
of  such  ODR  problems  that  have  arisen  at  NIST  and  that  exempbfy  the  performance  of 
ODRPACK.  P.  Boggs  will  also  provide  examples  of  ODRPACK’s  use,  showing  the  results 
of  simulation  studies  that  compare  ODR  to  OLS  in  a  controlled  setting.  The  ODRPACK 
software  will  be  available  for  demonstration  and  distribution. 
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Real  World  Applications  of  Total  Least 
Squares:  Direction-finding  and  System 

Identification. 

Richard  Roy 

Systems  Research  Associates,  Inc., 

10030  Phar  Lap  Drive,  Cupertino,  CA  95014,  U.S.A. 


Various  aspects  and  experiences  in  the  application  of  TLS  concepts  to  the  problems 
of  direction-of- arrival  (DOA)  estimation  and  multivariable  linear  system  identification  will 
be  discussed.  A  demonstration  of  severtd  such  applications  to  real  data  will  be  presented. 
These  will  include  DOA  estimation  in  a  coherent  source  environment,  and  system  identi¬ 
fication  of  a  large  flexible  space  structure.  A  MATLAB  Toolbox  for  sensor  array  signal 
processing  (AP  Toolbox)  will  be  presented  along  with  a  similar  set  of  routines  for  system 
identification.  Some  open/difficult  problems  remaining  to  be  solved  will  be  described  and 
recent  attempts  at  their  solution  outlined.  These  include  exploitation  of  problem  structure 
to  reduce  computational  complexity,  and  the  problem  of  detecting  dimensionality  of  the 
appropriate  subspaces  in  these  problems. 
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Tensorial  Calibration  in  Chemistry. 


Mary  B.  Seasholtz  and  Brure  R.  Kowalski 

Center  for  Process  Analytical  Chemistry, 

University  of  Washington, 

BG-10,  UW,  Seattle,  WA  98195,  U.S.A. 


One  goal  in  analytical  chemistry  is  to  relate  a  series  of  readily  acquired  measurements 
to  a  physical  or  chemiceil  property  of  a  sample.  Tensorial  calibration  is  a  context  for  char¬ 
acterizing  the  types  of  data  produced  by  analytical  instrumentation.  Instruments  measure 
(per  sample)  a  response  that  is  either  a  zeroth,  first,  second  or  third  order  tensor  (scalar, 
vector,  matrix  or  cube,  respectively).  The  Total  Least  Squares  (TLS)  problem  arises  in 
first  order  calibration,  when  a  vector  of  data  is  coUected  per  sample,  giving  the  model 
c  =  Rh  +  e,  where  c  is  a  vector  containing  the  concentrations  for  all  the  samples,  i?  is  a 
matrix  with  the  response  for  each  sample  in  a  row,  and  h  is  the  regression  vector. 

A  method  called  Parsimonious  Regression  (PR)  will  be  presented  that  reduces  the 
amount  of  measurement  error  in  R  incorporated  into  the  model.  This  method  starts  with 
a  principal  component  decomposition  of  R.  Even  the  largest  principal  component  factors 
(eigenvectors  of  RRP  and  RF R  with  large  eigenvalues)  will  contain  some  measurement 
noise.  This  method  eliminates  one  dimension  (rank)  at  a  time  according  to  a  cross  val¬ 
idation  criterion,  which  forces  the  parts  of  the  eigenvectors  corresponding  to  nonrelevant 
variance  (measurement  error)  to  be  removed.  Cross  validation  more  closely  models  the 
predictive  capability  of  a  model  than  does  the  fit  of  the  data  used  to  make  the  model,  and 
will  not  allow  correlations  of  noise  in  iZ  to  c  to  be  incorparated  into  the  model.  The  one 
remaining  dimension  after  the  decomposition  has  been  reduced  to  rank  one  and  will  point 
in  the  direction  of  the  regression  vector  6.  This  method  will  be  compared  to  principal 
component  regression  (PCR)  and  partial  least  squares  (PLS).  PCR  and  PLS  minimize  a 
least  squares  fit  criterion  with  a  single  step  projection,  while  in  PR  any  criterion  can  be 
minimized. 
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Errors  in  System  Identification. 


Torsten  Soderstrom 

Automatic  Control  &  Systems  Analysis  Group, 

Department  of  Technology,  Uppsala  university, 
P.O.Box  27,  75103  Uppsala,  Sweden 


When  applying  system  identification  (modelling  of  dynamic  systems)  it  is  necessary  to 
account  for  disturbances  and  measurement  errors.  Some  different  cases,  like  presence  of 
input  and  output  errors,  are  discussed,  as  well  ais  their  consequences  for  parameter  fitting. 
It  is  demonstrated  how  a  standard  least  squares  fit  does  not  give  a  consistent  model  when 
the  output  measurement  noise  is  uncorrelated  (white  noise).  A  bias  compensp'  .n  can  be 
used  at  a  modest  cost  to  give  a  consistent  model.  The  method  so  construct  an  also 
be  interpreted  as  a  classical  eigenvector  method  or  a  total  leeist  squares  fit.  Its  statistical 
properties  are  reviewed.  The  parameter  estimates  are  consistent  and  asymptotically  Gaus¬ 
sian  distributed.  Alternative  identification  methods,  like  instrumental  variables  and  output 
error  methods,  are  briefly  discussed,  and  compared  to  the  total  least  squares  approach. 
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Updating  Rank  Revealing 
URV  Decompositions. 


Gilbert  W.  Stewart 

Department  of  Computer  Science, 

Institute  for  Advanced  Computer  Studies, 

University  of  Maryland,  College  Park,  MD  207^2,  U.S.A. 

Let  A'  be  an  n  •  p  matrix  and  let  x  be  a  p-vector.  In  many  applications  it  is  required 
to  determine  the  effective  rank  of 


and  compute  an  approximate  null  space  for  A'.  In  general,  the  problem  will  have  already 
been  solved  for  A'  by  means  of  a  suitable  decomposition.  The  problem  is  then  to  update 
this  decomposition  to  obtain  the  corresponding  decomposition  of  X. 

In  many  ways  the  decompositions  of  choice  are  the  singular  value  decomposition  of 
A'  or  the  spectral  decomposition  of  C  =  A'^A'.  However,  b^th  these  decompositions  are 
difficult  to  update.  An  alternative  is  the  (pivoted)  QR  decomposition  of  X  or  the  Cholesky 
decomposition  of  C ,  both  of  which  can  be  updated.  However,  these  decomposition  do  not 
provide  an  explicit  basis  for  the  approximate  null  space. 

In  this  talk  we  describe  a  class  of  orthogonal  decompositions,  called  URV  decomposi¬ 
tions.  which  in  some  sense  lie  between  the  singular  value  decomposition  and  the  QR  de¬ 
composition  and  share  the  advantages  of  both.  They  can  be  made  rank  revealing  and  they 
furnish  an  explicit  basis  for  the  approximate  nuU  space.  Moreover,  they  can  be  updated 
sequentially  in  0{p^)  time.  On  a  linear  array  of  p  processors,  they  can  be  updated  in  0(p) 
time. 


Comparative  Performance  Study  of  Least 
Squares  and  Total  Least  Squares 
Yule- Walker  Estimates  of  Autoregressive 

Parameters. 

Adina  Stoian 

General  Direction  for  Telecomuunicoiions, 

SOS.  Colenttna  nr.  22,  BL.9C  SC.C  ET.3, 

AP.IO4,  Sector  2,  Bucharest-72264,  Ro  ^mania 


This  talk  considers  the  estimation  of  the  parameters  of  autoregressions  from  noisy  mea¬ 
surements,  A  new  method  for  solving  this  problem  is  presented,  which  uses  a  combination 
of  two  recent  algorithms:  a  High-Order  Yule-Walker  (H^^YW)  algorithm  for  estimation  of 
the  autoregressive  (AR)  parameters  of  an  autoregressive  moving  average  (ARM A)  model, 
introduced  by  P.  Stoica,  B.  Friedlander  and  T.  Soderstrom,  and  the  Total  Least  Squares 
(TLS)  algorithm  studied  by  S.  Van  Huffel.  Simulation  results  are  presented  in  order  to 
evaluate  the  proposed  HOYW-TLS  technique  and  the  HOYW-LS  technique  with  respect 
to  their  accuracy  in  parameter  estimation  of  autoregressions.  The  performances  of  these 
two  techniques  are  mainly  influenced  by  the  following  factors:  the  pole-zero  configuration 
of  the  ARMA  processes,  the  sample  length  and  the  dimension  of  the  HOYW  system.  It  is 
concluded  that  the  HOYW-TLS  technique  is  especially  useful  for  any  sample  length  and 
any  dimension  of  the  HOYW  systen.,  when  the  zeros  of  the  ARMA  model  are  close  to  the 
unit  circle.  However,  even  if  no  information  about  the  singularities  of  the  ARMA  process 
is  available,  the  use  of  the  TLS  method  is  proved  to  be  strongly  recommended,  for  it  is  in 
practice  never  less  accurate  than  the  classical  LS  technique. 
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Signal  Enhancement  Motivated  by  Total 
Least  Squares  and  Linear  Prediction. 


Donald  W.  Tufts  and  Abhijit  A.  Shah 
Department  of  Electrical  Engineering, 

University  of  Rhode  Island, 
Kingston,  RI-02881,  U.S.A. 


Evaluation  and  improvement  of  a  SVD-based  data-adaptive  signal  estimation  algorithmfl] 
are  presented.  In  the  first  step  of  this  algorithm  a  matrix  with  a  Hankel-like  structure  is 
formed  from  the  noise  corrupted  time-series  data.  It  is  assumed  that  the  rank  of  the  signal 
only  data  matrix  is  known  or  is  obtainable  from  the  Singular  Vcdue  Decomposition  (SVD) 
of  the  noise  corrupted  data  matrix.  What  is  important  is  that  the  signed  only  matrix  can  be 
well  approximated  by  a  matrix  of  lower  rank.  Using  SVD  such  a  low  rank  approximation 
of  the  signal  only  matrix  is  made.  By  summing  across  the  diagonals  of  this  “cleaned  up” 
post-SVD  low  rank  matrix,  time  series  data  with  enhanced  signal  component  is  extracted. 

The  highly  nonlinear  nature  of  SVD  makes  performance  evaluation  of  such  SVD-based 
applications  a  difficult  task,  but  methods  of  backward  error  analysis  make  the  task  easier. 
The  best  low  rank  approximation  of  a  perturbed  matrix  is  approximated  by  a  first  order 
perturbation  expansion  of  the  orthogonal  subspace.  This  makes  computation  of  statistical 
properties  of  the  post-SVD  data  a  tractable  problem.  Using  the  obtained  statistical  infor¬ 
mation.  performance  of  the  signal  enhancement  algorithm  is  evaluated. 

It  is  also  shown  that  this  statistical  information  can  be  utilized  to  extract  the  en¬ 
hanced  signal  from  the  post-SVD  low  rank  matrix  by  an  optimal  linear  combination  of  its 
elements.  Computer  simulation  results  for  verifying  theoreticed  predictions  and  for  com¬ 
paring  the  performance  of  the  original  and  modified  algorithms  to  the  Cramer-Rao  bound 
are  presented. 


References 
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Asymptotic  Properties  of  a  Class  of 
Regression-type  Estimators. 


Hugo  Van  hamme  and  Rik  Fintelon 
division  ELEC,  Department  of  Electrical  Engineering, 

Vrije  Universiteit  Brussel, 

Pleinlaan  2,  1050  Brussel,  Belgium 


An  asymptotic  analysis  of  the  regression  problem  Hp  =  0  where  only  a  noisy  observa¬ 
tion  G  of  is  available  is  presented.  A  class  of  estimators  for  p  is  given  by  the  minimizer 
of  p^G^ Dp^Gp  over  p,  where  Dp  is  a  positive  semi-definite  Toeplitz  weighting  matrix  with 
entries  that  are  quadratic  in  p.  Particular  choices  of  Dp  result  in  the  maximum  likeli¬ 
hood  estimator  (MLE)  if  the  noise  G  —  H  is  zero-mean  Gaussian  with  (partially)  known 
covariance,  an  approximate  MLE  (AMLE)  or  the  generalized  totid  least  squares  (GTLS) 
estimator. 

Sufficient  conditions  for  consistency  and  asymptotic  normality  for  non-Gaussian  noise 
are  given  in  terms  of  absolute  summability  of  joint  cumulants  and  constraints  on  the 
weighting  matrix  Dp.  Expressions  for  the  asymptotic  parameter  covariance  are  given  and 
show  that  the  MLE  doesn’t  reach  the  Cramer-Rao  bound. 
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The  Total  Least  Squares  Problem  :  current 
state  of  research  and  applications. 

S^.bine  Van  HuiTei’ 

Katholieke  Universiteit  Leuven, 

ESAT  Laboratory,  Department  of  Electrical  Engineering, 
Kardinaal  Mercierlaan  94,  B-3001  Heverlee,  Belgium 


The  aim  of  this  lecture  is  to  present  an  overview  of  the  current  state  of  research  of 
a  modeling  technique  which  is  known  as  total  least  squares  in  numerical  linear  algebra 
and  signal  processing,  errors-in-variahles  and  orthogonal  regression  in  the  statistical  com¬ 
munity,  eigenvector  or  Koopmans-Levin  method  and  compensated  least  squares  in  system 
identification. 

The  basic  motivation  for  Total  Least  Squares  (TLS)  is  the  following:  Let  a  set  of 
multidimensional  data  points  (vectors)  be  given.  How  can  one  obtain  a  linear  model  that 
explains  these  data?  The  idea  is  to  modify  all  data  points  in  such  a  way  that  some  norm 
of  the  modification  is  minimized  subject  to  the  constraint  that  the  modified  vectors  satisfy 
a  linear  relation. 

The  origin  of  this  basic  insight  can  be  traced  back  to  the  beginning  of  this  century.  It 
was  rediscovered  many  times,  often  independently,  nieiinly  in  the  statistical  and  psychome¬ 
tric  literature.  However,  it  is  oidy  in  the  leist  decade  that  the  technique  also  penetrated  in 
scientific  and  engineering  applications.  One  of  the  main  reasons  for  its  sudden  popularity 
is  the  availability  of  efficient  and  numerical  robust  algorithms,  in  which  the  singular  value 
decomposition  plays  a  prominent  role.  Another  reason  is  the  fact  that  TLS  is  really  an 
applications  oriented  procedure:  It  is  ideally  suited  for  situations  in  which  all  data  are 
corrupted  by  noise,  which  is  almost  always  the  case  in  engineering  applications.  In  this 
sense  it  is  a  powerful  extension  of  the  classical  least  squares  idea,  which  corresponds  only 
to  a  partial  modification  of  the  data. 

This  lecture  surveys  the  TLS  problem  and  enlightens  the  main  results.  In  particular, 
the  computational,  as  well  as  the  numerical  and  statistical  aspects  of  the  TLS  problem  are 
discussed  and  compared  with  those  of  other  related  problems,  e.g.  LS  and  multivariate 
regression  techniques.  Different  TLS  algorithms  are  presented  and  evaluated  and  practi¬ 
cal  guidelines  are  given  which  delineate  the  domain  of  applicability  of  the  TLS  method. 
Furthermore,  some  recent  generalizations  of  the  TLS  problem  are  presented,  as  well  as  an 
overview  of  its  applications. 


^Senior  Research  Assistant  of  the  Belgian  N.F.W.O.  (National  Fund  for  Scientific  Research) 
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Solving  Totd  Least  Squares  (TLS)  problems  AX  ==  B  requires  the  computation  of 
the  noise  subspace  of  the  data  matrix  [A-,  B].  The  widely  used  tool  for  doing  this  is 
the  Singular  Value  Decomposition  (SVD).  However,  the  SVD  has  the  drawback  that  it  is 
computationally  expensive.  Therefore,  we  consider  here  a  different  so-called  rank- revealing 
two-sided  orthogonal  decomposition  which  decomposes  the  matrix  into  a  product  of  a 
unitary  matrix,  a  triangular  matrix  and  another  unitary  matrix  in  such  a  way  that  the 
effective  rank  of  the  matrix  is  obvious  and  at  the  same  time  the  noise  subspace  is  exhibited 
explicitly.  We  show  how  this  decomposition  leads  to  an  efficient  and  reliable  TLS  algorithm 
that  can  be  parallelized  in  an  efficient  way. 


^Senior  Research  Assistant  of  the  Belgian  N.F.W.O.  (National  Fund  for  Scientific  Research) 
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Robust  Estimation  in  Measurement  Error 

Models. 
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This  talk  will  discuss  the  properties  of  robust  estimation  for  measurement  error  mod¬ 
els.  A  method  for  converting  ordinary  regression  algorithms  to  orthogonal  regression  algo¬ 
rithms  will  be  described.  The  main  discussion  will  primarily  consider  bounded  influence 
estimation  using  generalized  M-estimators.  Topics  will  include  sharp  lower  bounds  on  the 
sensitivities,  most  bounded  robust  estimators  and  Hampel’s  optimality  problem.  The  lat¬ 
ter  finds  estimators  which  minimize  the  asymptotic  variance  subject  to  an  upper  bound 
on  the  influence  function. 
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Total  Chebyshev  Approximation 


G.  Alistair  Watson 

Department  of  Mathematics  and  Computer  Sciences, 
University  of  Dundee,  Dundee,  DDI4HN,  Scotland,  U.K. 


A  generalization  of  the  toted  least  squares  problem  to  infinite  dimensional  spaces  is 
considered,  which  leads  naturally  to  the  total  Chebyshev  approximation  problem: 

n 

minimizejl  ^  ai</>i(i)|loo  subject  to  ||al|2  =  1, 

t=i 

where  the  functions  4>i{x)  are  given  real  functions.  Various  cispects  of  this  problem  are 
considered. 
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Shanghai  200062,  China 


This  talk  presents  an  analysis  of  the  solutions  of  the  Total  Least  Squares  (TLS)  problem 
AX  ^  B  in  cases  where  the  matrix  [A;  B]  may  have  multiple  smallest  singular  values. 
Genercd  formulas  for  the  minimum  norm  TLS  solutions  are  given;  the  difference  between 
the  TLS  and  the  LS  solutions  is  obtained;  the  error  bounds  for  the  perturbed  TLS  solutions 
with  or  without  minimal  length  are  deduced.  The  analysis  is  useful  especially  for  rank- 
deficient  problems  and  generalizes  the  results  of  Golub  and  Van  Loan,  Van  Huffel  and 
Vandewalle  and  Zoltowski.  Numericzd  results  for  a  practical  application  are  also  given  to 
verify  the  error  bounds. 


46 


WORKSHOP  ON 

“Total  Least  Squares;  Concepts,  Algorithms, 

Applications”. 

August  12-15,  1991. 


LIST  of  PARTICIPANTS: 


names,  affiliations  and  addresses 


47 


Dr.  Theagenis  J.  Abatzoglou 

Lockheed  Missiles  &  Space,  Inc. 

Research  &  Development  division  (0/9150,8251) 
3251  Hanover  Street 
Palo  Alto,  CA  94304-1191 
U.S.A. 

agis@galaxy.  decnet  .lockheed.  com 

Prof.  Larry  P.  Ammann 
Programs  in  Mathematical  Sciences 
University  of  Texeis  at  Dallas 
Richardson,  TX  75080 
U.S.A. 

ammann@utdallas.edu 


Wouter  Beck 

Fysisch  en  Elektronisch  Laboratorium  TNO 

P.O.  Box  96964 

2509  JG  Den  Haag 

The  Netherlands 

wbeck@fel.tno.nl 


Prof.  Paul  T.  Boggs 

National  Institute  of  Standards  and  Technology 
Applied  and  Computioned  Mathematics  Division 
Building  225,  Room  A151 
Gaithersburg,  Maryland  20899 
U.S.A. 

boggs@cam.nist.gov 

Prof.  James  R.  Bunch 

Department  of  Mathematics 
University  of  California  at  San  Diego 
La  Jolla,  CA  92093-0112 
U.S.A. 

jbunch@ucsd.edu 

Dr.  Chi-Lun  Cheng 
Institute  of  Statistical  Science 
Academia  Sinica 
Tjupei 

Taiwan,  R.O.C. 

STCHEN®TWNAS886.bitnet 


Prof.  Yasuo  Amemiya 

Department  of  Statistics 
Iowa  State  University 
102G  Snedecor  Hall 
Ames,  Iowa  50010 
U.S.A. 

Prof.  K.S.  Arun 

Coordinated  Science  Laboratory 
University  of  Illinois 
1101  West  Springfield  Avenue 
Urbana-Champaign,  IL  61801 
U.S.A. 

au'un@uicsl. csl.uiuc.edu 

Prof.  A.  BJorck 

Department  of  Mathematics 
Linkoping  University 
S-581  83  Linkoping 
Sweden 

akbJo%math. liu.se@elrond.ida.liu.se 

Karl  S.  Booksh 

Center  for  Process  Analytical  Chemistry 
University  of  Washington  BG-10,  UW 
Seattle,  WA  98195 
U.S.A. 

booksh@  cpac .  Washington,  edu 

Prof.  James  A.  Cadzow 
Department  of  Electrical  Engineering 
P.O.Box  6080,  STA.  B 
Vanderbilt  University 
Nashville,  Tennessee  37235 
U.S.A. 

jac@vuse.vanderbilt.edu 

Dr.  Kenneth  D.  Clark 

Mathematical  Sciences  Division 
Army  Reseuch  office 
Research  Triangle  Park 
NC  27709-2211 
U.S.A. 

clark@adm.csc.ncsu.edu 


48 


Johan  David 

ESAT  Laboratory,  Dept,  of  Electrical  Engineering 

Katholieke  Universiteit  Leuven 

Kardinaal  Mercierlaan  94 

3001  Heverlee 

Belgiiun 

david@esat.kuieuven.ac.be 

Anne-Marie  De  Meyer 
Universitair  Rekencentrum 
Katholieke  Universiteit  Leuven 
Willem  De  Croylaian  52 
3001  Heverlee 
Belgium 

LAAAA14@BLEKUL21.bitnet 

Prof.  Blurt  De  Moor 

ESAT  Laboratory,  Dept,  of  Electrical  Engineering 

Katholieke  Universiteit  Leuven 

Kardinaal  Mercierlaan  94 

3001  Heverlee 

Belgium 

demoor@esat.kuleuven.ac.be 

Prof.  George  Fisher 

Department  of  Earth  and  Planetary  Sciences 
The  Johns  Hopkins  University 
Baltimore,  Maryland  21218 
U.S.A. 


Prof.  Gene  Golub 

Department  of  Computer  Sciences 
Stanford  University 
Stanford,  CA  94305 
U.S.A. 

golub@patience.stanford.edu 

Patrick  Guillaume 

Faculty  of  Applied  Sciences 
Department  ELEC 
Vrije  Universiteit  Brussel 
Pleinlaan  2 
1050  Brussel 
Belgium 


Pieter  De  Bie 

ESAT  Laboratory,  Dept,  of  Electriced  Engineering 

Katholieke  Universiteit  Leuven 

Kardinaal  Mercierlaan  94 

3001  Heverlee 

Belgium 

debie@esat.kuleuven.ac.be 

Prof.  James  Demmel 

University  of  California  at  Berkeley 
Computer  Science  Division 
Evans  Hcdl  513 
Berkeley,  CA  94720 
U.S.A. 

demmel@arpa.berkeley.edu 

Ricardo  Fierro 

Department  of  Mathematics 
University  of  Ccdifornia  at  San  Diego 
La  JoUa,  CA  92093-0112 
U.S.A. 

fierro@euler.ucsd.edu 


Prof.  Wayne  A.  Puller 

Department  of  Statistics 
Iowa  State  University 
Snedecor  Hall 
Ames,  Iowa  50010 
U.S.A. 


Prof.  Roberto  Guidorzi 

Dip.  di  Electtronica,  Informatica  e  Sistemistica 

University  of  Bologna 

Viale  del  Risorgimento  2 

40136  Bologna 

Italy 

guidorzi@ingbol.cineca.it 

Sven  Hammarling 

Niunerical  Algorithms  Group  Ltd. 

Wilkinson  House 
Jordan  Hill  Road 
Oxford  0X2  SDR 
United  Kingdom 
nagsven@uk.ac..  xford.vax 


49 


Dr.  Trevor  Hastie 

AT&T  Bell  Laboratories  2C-261 
Murray  Hill,  New  Jersey  07974 
U.S.A. 

trevor@research.att.  com 


Jae  Hak  Lee 

Department  of  Geology  and  Geophysics 
Yale  University 
P.O.  Box  6666 

New  Haven,  Connecticut  06511 
U.S.A. 

lee%stormy@  Venus. ycc.yale.edu 

Prof.  Franklin  Luk 

School  of  Electrical  Engineering 
Cornell  L^ruversity 
Ithaca,  NY  14853 
U.S.A. 

luk@jacobi.ee.cornell.edu 

Dr.ir.  Marc  Moonen 

ESAT  Laboratory,  Dept,  of  Electricad  Engineering 

Katholieke  Universiteit  Leuven 

Kardineial  Mercierlaan  94 

3001  Heverlee 

Belgium 

moonen@esat.kuleuven.ac.be 

Dr.ir.  Rik  Pintelon 

Faculty  of  Applied  Sciences 
Department  ELEC 
V^rije  LTniversiteit  Brussel 
Pleinlaan  2 
1050  Brussel 
Belgium 

Janet  E.  Rogers 

National  Institute  of  Standards  and  Technology 
Applied  and  Computational  Mathematics  Division 
325  Broadway, 

Boulder,  Colorado  80303-3328, 

U.S.A. 

jrogers@cam.nist.gov 


Prof.  Nick  Higham 

Department  of  Mathematics 
University  of  Manchester 
Manchester  M13  9PL 
United  Kingdom 
na.nhigham@na-net.ornl.gov 

Dr.ir.  Jan  Leuridan 

Leuven  Measurement  &  Systems  International 
Interleuvenlaan  65 
3001  Heverlee 
Belgium 


Jean-Franqois  Maitre 

Ecole  centrale  de  Lyon 
Departement  de  Mathematiques 
B.P.  163 

69131  Ecully  cedex 
FVeince 

Christiaan  Moons 

ESAT  Laboratory,  Dept,  of  Electrical  Engineering 

Katholieke  Universiteit  Leuven 

Kardinaal  Mercierlaan  94 

3001  Heverlee 

Belgium 

moons@esat  .kuleuven.ac  .be 

Dr.  ir.  Jose  Ramos 

ESAT  Laboratory,  Dept,  of  Electrical  Engineering 

Katholieke  Universiteit  Leuven 

Kardinaal  Mercierlaan  94 

3001  Heverlee 

Belgium 

ramos@esat.kuleuven.ac.be 

Yves  Rolain 

Faculty  of  Applied  Sciences 
Department  ELEC 
Vrije  Universiteit  Brussel 
Pleinlaan  2 
1050  Brussel 
Belgium 


50 


Dr.ir.  Richard  Roy 

Systems  Research  Associates,  Inc. 

10030  Phar  Lap  Drive 
Cupertino,  CA  95014 
U.S.A. 

dick@sprite.stanford.edu 

Prof.  Louis  Scharf 

Department  of  Electriceil  cind  Computer  Engineering 

University  of  Colorado 

Campus  Box  425 

Boulder  CO  80309-0425 

U.S.A. 

scharf%dsp@boulder. colorado.edu 

Mary  B.  Seasholtz 

Center  for  Process  Analytical  Chemistry 
University  of  Washington  BG-10,  UW 
Seattle,  WA  98195 
U.S.A. 

marybeth@cpac.washington.edu 

Prof.  Gilbert  W.  Stewart 

Department  of  Computer  Science 
University  of  Meuyland 
College  Park,  Maryland  20742 
U.S.A. 

stewart@cs.umd.edu 


Dr.  Werner  Stuetzle 

Department  of  Statistics,  GN-22 
University  of  Washington 
Seattle,  WA  98195 
U.S.A. 

wxs@stat.washington.edu 

Or.ir.  Paul  Van  den  Hof 

Dept,  of  Mechanical  Engineering  &  Marine  Tcchn. 

Technische  Universiteit  Delft 

Mekelweg  2 

2628  CD  Delft 

The  Netherlands 

vdhof@tudw03.tudelft.nl 


Dr.  Aca**  Savaci 

ESAT  Laboratory,  Dept,  of  Electrical  Engineering 

Katholieke  Universiteit  Leuven 

Kardinaal  Mercierl£ian  94 

3001  Heverlee 

Belgium 

savaci@esat.kuleuven.ac.be 

Prof.  Johan  Schoukens 

Faculty  of  Applied  Sciences 
Department  ELEC 
Vrije  Universiteit  Brussel 
Pleinlaan  2 
1050  Brussel 
Belgium 

Prof.  Torsten  Soderstrdm 

Automatic  Control  &  Systems  Analysis  Group 

Department  of  Technology 

Uppsala  University,  P.O.  Box  27 

75103  Uppscda 

Sweden 

ts@csig.sunet.se 

Adina  Stoian 

Generad  Direction  for  Telecommunications 

SOS.  Colentina  nr.22 

BL.9C  SC.C  ET.3,  AP.104,  Sector  2 

Bucharest- 72264 

Roumania 


Prof.  Donald  W.  Tufts 

Department  of  Electrical  Engineering 
University  of  Rhode  Island 
Kelley  Hall  Annex,  Room  A- 109 
Kingston,  RI  02881 
U.S.A. 

tufts@quahog.uri.edu 

Prof.  Joos  Var.dewalle 

ESAT  Laboratory,  Dept,  of  Electrical  Engineering 

Katholieke  Universiteit  Leuven 

Kardinaal  Mercierlaan  94 

3001  Heverlee 

Belgium 

vandewalle@esat.kuleuven.ac.be 


51 


Hugo  Van  hamme 

Faculty  of  Applied  Sciences 
Department  ELEC 
Vrije  Universiteit  Brussel 
Pleinleian  2 
1050  Brussel 
Belgium 

Prof.  John  Van  Ness 
Programs  in  Mathematical  Sciences 
University  of  Texas  at  Dallas 
Richardson,  TX  75080 
U.S.A. 

ness@utdallas.edu 

Prof.  G.  Alistair  Watson 

Department  of  Mathematics  and  Computer  Science 

University  of  Dimdee 

Dundee  DD14HN 

Scotland,  U.K. 

gawatson@mcs.dund.ac.uk 


Dr.ir.  Sabine  Van  Huffel 

ESAT  Laboratory,  Dept,  of  Electriccd  Engineering 

Katholieke  Universiteit  Leuven 

Kardinaal  Mercierlaan  94 

3001  Heverlee 

Belgium 

vanhufFel  @esa  t .  kuleu  ven  .ac.be 

Magda  Vuylsteke 

Universiiair  Rekencentrum 
Katholieke  LTniversiteit  Leuven 
Willem  De  Croylaan  52 
3001  Heverlee 
Belgium 

LAAAA02@BLEKUL21.bit.iet 

Prof.  Musheng  Wei 

Department  of  Mathematics 
East  China  Normal  University 
Shcinghai  200062 
China 


S.  Wiersma 

Koninklijke  Shell-Laboratorium 

Afdeling  MSE 

P.O.Box  3003 

1003  AA  Amsterdam 

The  Netherlands 


Dr.  Kai-Bor  Yu 

GE  Research  and  Development  Center 
KW-C606,  P.O.Box  8 
Schenectady,  N.Y.  12301 
U.S.A. 

yu@crd.ge.com 


52 


